Head Mounted Eye Tracking Device and Method for Providing Drift Free Eye Tracking Through a Lens System

ABSTRACT

The invention relates to a method and a head mounted eye tracking device ( 10   a,    10   b ) for determining at least a feature of an eye ( 12 ), wherein the eye tracking device ( 10   a,    10   b ) comprises a capturing device (C) for capturing light reflected by the eye ( 12 ) and an optical component ( 14 ; E, E 1 , En, E 2 ) capable of altering a propagation direction of the light passing through the optical component ( 14 ; E, E 1 , En, E 2 ). The eye tracking device ( 10   a,    10   b ) is configured such that when fixed to the head of the user the light captured by the capturing device (C) has passed through the optical component ( 14 ; E, E 1 , En, E 2 ) and constitutes at least part of an image and the feature is determined on the basis of the image in dependency of an information about a relative position between the eye ( 12 ) and the head mounted eye tracking device ( 10   a,    10   b ).

The invention relates to a head mounted eye tracking device fordetermining at least one first feature of at least one eye, wherein thehead mounted eye tracking device comprises at least one capturing devicefor capturing light reflected by at least one eye of a user and at leastone optical component capable of altering a propagation direction oflight passing through the optical component. Furthermore, the inventionrelates to a method for determining at least one first feature of atleast one eye of a user by means of a head mounted eye tracking device.

There are head mounted eye tracking devices known from the prior art,which can comprise a frame that can be mounted to the head of a user andlenses inserted into the frame through which the user can look. Usuallyeye tracking devices comprise cameras which capture images of the eyesof a user and therefrom determine the gaze direction. These cameras canbe placed in the frame and have a direct path to the user's eye. Suchconfigurations have the disadvantage that cameras only have a side viewonto the eye, at least for certain gaze directions, and eye features aremore difficult to capture. This reduces the tracking accuracy.Furthermore, these cameras can be seen by the user and might bedisturbing or occluding parts of the visual field.

Also there are known head mounted displays with integrated eye trackingdevices. Head mounted displays usually comprise a display, which ismounted to the head of a user by means of a frame, for example. Imagesshown on the displays can be projected onto the eyes of a user by meansof lenses or lens systems. Also there are head mounted devices knownfrom the prior art, in which the camera is hidden from the user and isobserving the user's eye through a lens system. These devices useappearance based, interpolating eye tracking approaches, which have thebig drawback that the calibration of these eye tracking devices isinvalidated when the relative position of the device with respect to theuser's eye is changed, for example if the head mounted eye trackingdevice slips slightly. This results in a loss of accuracy aftermovement.

Therefore it is the object of the present invention to provide a headmounted eye tracking device and a method for determining at least onefirst feature of at least one eye of a user by means of a head mountedeye tracking device, which provides more flexibility in positioningcomponents of the head mounted eye tracking device and at the same timeavoids detrimental impact on the tracking accuracy.

This object is solved by a head mounted eye tracking device and a methodfor determining at least one feature of at least one eye of a user bymeans of a head mounted eye tracking device with the features of theindependent claims. Advantageous embodiments of the invention arepresented in the dependent claims.

According to the invention the head mounted eye tracking device isconfigured such that, when fixed to the head of a user, the lightcaptured by the capturing device has passed through the at least oneoptical component before and/or after it had been reflected by the atleast one eye of the user and constitutes at least part of an image.Thereby, the eye tracking device is configured to determine the at leastone first feature on the basis of the image and in dependency of aninformation about a relative position between the at least one eye andthe head mounted eye tracking device.

So advantageously, a change in a relative position between the at leastone eye and the head mounted eye tracking device can be taken intoaccount when performing eye tracking, so that no calibration has to berepeated when a change of the relative position takes place and at thesame time the eye tracking quality is not negatively influenced.Furthermore, as the captured image is constituted at least at part bylight that has passed through the optical component this set up allowsfor much more flexibility in positioning the components of the headmounted eye tracking device. Thereby, the capturing device, which cancomprise one or more cameras or image sensors, and/or light sources ofthe head mounted eye tracking device, if any, can be positioned behindthe optical component with regard to the viewing direction of the user.This additionally allows for hiding these other components from the userin very beneficial way. Moreover, this has the great advantage thatthese other components can be optimized in their positions so that ahigher tracking quality and accuracy can be achieved. Additionally thisallows for positioning the optical component closer to the eyes of theuser so that also the visual perception of the surrounding of the useror images shown on a display device of the eye tracking device can beimproved.

The information about the relative position between the eye and the headmounted eye tracking device moreover allows for a drift freelocalization of the at least one first feature of the at least one eye,even when components of the head mounted eye tracking device, like lightsources or the capturing device, are positioned behind the opticalcomponent. Moreover, the relative position between the at least one eyeand the head mounted eye tracking device is to be understood as therelative position between the eye as a whole and the eye trackingdevice. The relative position between the eye tracking device and thehead of the user would be an equivalent characterization, so thatinstead of the eye tracking device being capable to determine the atleast one first feature on the basis of the image and in dependency ofan information about a relative position between the at least one eyeand the head mounted eye tracking device one could also say that the eyetracking device is capable of determining the at least one first featureon the basis of an information about a relative position between thehead of the user and the head mounted eye tracking device.

In general, the at least one optical component can comprise and/or canbe configured as a lens, a prism, a spherical lens, an aspherical lens,a free-form lens, a waveguide, a lens system, or an optical systemcomprising one or more of the named lenses and/or prisms. The at leastone optical component also can comprise and/or can be configured as anycombination of above named elements and/or systems. Especially, the atleast one optical component comprises a refractive optical componentand/or a reflective optical element. Such imaging optics relate objectsand images mathematically, so that for example the information about therelative position between the eye and the tracking device can also bederived from the image itself when knowing the mathematical relation oran approximation thereof or in general at least one information aboutthe optical component. Moreover, the head mounted eye tracking devicecan comprise one or more further optical components additionally to theat least one optical component. These further optical components canalso be configured as or comprise above named lenses, prism, reflectiveoptical elements like mirrors, and so on.

Furthermore, it is preferred that the optical component is positioned infront of the eye. So the user can see through the optical component andthe optical component can project the user's surrounding or an image ona display onto his/her eyes. This is especially advantageous if the headmounted eye tracking device at the same time is configured as a headmounted display or in other words also comprises a display fordisplaying images to the at least one eye.

Therefore it is a preferred embodiment of the invention that the eyetracking device comprises a display device configured to display images,preferably wherein the head mounted eye tracking device is configuredsuch that light originating from the display device at least in partpasses through the at least one optical component and impinges on the atleast one eye, especially when the head mounted eye tracking device isfixed to the head of the user. So the optical component canadvantageously be used to project images shown on the display ordisplays into almost the complete field of view of the user.Furthermore, the display can be configured as so called bidirectionaldisplay, which is configured to display and capture images at the sametime. Therefore, the display of the head mounted eye tracking device cancomprise and/or be configured as the capturing device of the headmounted eye tracking device, which facilitates an especially compactconfiguration of the head mounted eye tracking device.

Moreover, the information about the relative position can be derived inseveral ways. There are advantageous and simple ways to derive thisinformation directly from the captured images. To do so, it isbeneficial to know something about the optical component, like itsimaging properties and/or above named mathematical relation and/orapproximation thereof. This information can be defined and be known tothe head mounted eye tracking device, e.g. stored in a storage device ofthe eye tracking device or a storage device the eye tracking device canbe coupled with and/or implemented in an algorithm performed by aprocessing unit of the eye tracking device for eye tracking, or the headmounted eye tracking device can be configured to derive thisinformation, for example, by performing a calibration procedure. Theinformation about the relative position can also be determinedotherwise, for example without knowing anything about the opticalcomponent. The head mounted eye tracking can be configured such thatimages of the eye and/or face of the user can be captured that comprisedfeatures that do not change in their position relative to the eyetracking device when the user changes his/her gaze direction, except therelative position between the user and the eye tracking device changes.For example, eye corners of the eye can be detected. If the eye trackingdevice determines the position of the pupil based on the capturedimages, this position changes in the images if the user changes his/hergaze direction, but the position of the eye corners does not change dueto changes in the gaze direction. So it can be determined whether thechange of a position of the pupil is due to a movement of the eyetracking device relative to the user's eye, because in this case theposition of the eye corner would have changed as well, or the change inthe position of the pupil is due to the user changing his/her directionof view, because in this situation, the position of the eye corner doesnot change. Instead of eye corners, also other features of the eye orface can be detected, like eyebrows, the nose or the like. The headmounted eye tracking device can also comprise a component for detectingthe information about a relative position, like a change in the relativeposition, for example, a sensor, especially a proximity sensor, anaccelerometer, or an inertial measurement unit.

Consequently, there are many simple and favorable ways to determine theinformation about the relative position between the eye and the eyetracking device and to use this information for correcting or rescalingdetermined eye features.

Besides, the head mounted eye tracking device can be configured toperform model based eye tracking and/or appearance based eye tracking.In both cases it is advantageously possible to determine the at leastone feature of the at least one eye also if the capturing device and/orlight sources are positioned behind the optical component without havinga negative influence on the eye tracking quality.

Furthermore, the head mounted eye tracking device can be configured todetermine the at least one first feature as at least one of the group ofa pupil, a sclera, a limbus, an iris, a cornea, a cornea center, aneyeball center, a pupil center, a pupil diameter, a pupilcharacteristic, a limbus characteristic, a sclera characteristic, aniris characteristic, a characteristic of a blood vessel, a corneacharacteristic, an eyeball characteristic, a gaze direction, a point ofgaze, an orientation, a position and an eyelid closure of the at leastone eye. Also more of these features can be determined, for example, todetermine the gaze direction first the eyeball center and the pupilcenter can be determined and therefrom a gaze vector can be calculated.Also other combinations of the features of above named group or singlefeatures of above named group can be used to calculate the gazedirection vector.

In this context the point of gaze is the point a user is looking at. Itcan be calculated by intersecting the gaze direction vectors of each eyeof the user. If a two dimensional surface the user is looking at isknown, e.g. the person is looking at a screen or display that may beintegrated into the head mounted eye tracking device, the point of gazecan be calculated as the intersection of the gaze direction of the lefteye or of the right eye or of an averaged gaze direction of the left andthe right eye with this surface. Also more advanced methods forcalculating the binocular point of gaze may be used, e.g. taking intoaccount the users physiognomy or conditions such as nystagmus orstrabismus in a statistical, stochastic or higher level model.

In an advantageous embodiment of the invention the eye tracking deviceis configured to perform a localization of a second feature of the atleast one eye in dependency of the information about the relativeposition between the eye and the eye tracking device based on the imageand to determine the at least one first feature on the basis of thelocalization of the at least one second feature. As already mentioned,to determine the gaze direction as first feature, second features like apupil, cornea or eyeball, can be determined to do so. In general thesecond features can also be at least one in the above named group.

A localization of a second feature in this context can be understood asthe localization of the second feature in real space or as thelocalization of the second feature in the image. For example, if the eyetracking device performs appearance based or interpolating eye tracking,usually in a calibration procedure the eye tracking device capturesseveral images of the eyes of the user while he/she is looking atpredefined calibration points and the eye tracking device establishes amapping that maps for example the pupil position in the image to acorresponding point of gaze on a display.

Thereby no localization of the pupil in real space has to be performed.Now it is very advantageous to take into account the information aboutthe relative position between the eye and the eye tracking device,because then the eye tracking device is capable of differentiating thetwo situations whether the position of the pupil in the image haschanged due to a change in gaze direction or due to a movement of theeye tracking device relative to the user's head. To discern these bothsituations eye features like eye corners can be detected additionally asalready mentioned.

Furthermore, the localization of the second feature can also beperformed as a localization of the second feature in real space bydetermining two or three real space coordinates of the second featurewith respect to the eye tracking device.

Thereby it is very beneficial when the eye tracking device is configuredto perform the localization as a 3D localization. Thereby the positionof the eye, the position of a cornea or a position of the pupil or othereye features can be determined very accurately, so that also the gazedirection can be calculated precisely.

This 3D localization is preferably performed in conjunction with modelbased eye tracking. Therefore, it is another very advantageousembodiment of the invention when the head mounted eye tracking device isconfigured to determine the at least one first feature of the at leastone eye based on a model of the at least one first and/or at least onesecond and/or at least one third feature of the at least one eye. Forexample the eye orientation can be inferred from the perspectivedeformation of the pupil contour or limbus contour and a distance to therespective eye can be calculated from the dimensions in the image of eyefeatures which do not change dynamically, such as limbus major and minoraxis, or eyeball radius. So for example, if the eyeball radius isassumed to be the same for every user and is known, by determining theeyeball radius from an image and by setting this determined radius inrelation to the actual known radius the distance of the eye from thecamera or the capturing device in general can be determined. For thecase that the camera is seeing the eye through the at least one opticalcomponent the optical properties of the optical component have to betaken into consideration. If the focal length, a scaling factor or anyother imaging property of the optical component is known, forms anddimensions detected in the captured image can be mapped to thecorresponding real forms and dimensions. Therefrom e.g. the 3D-positionand/or 3D-orientation of the eye or other eye features can be derivedand by this a 3D-localization of these features is performed.

Consequently, it is very advantageous that according to anotherembodiment of the invention the head mounted eye tracking device isconfigured to determine the at least one first feature and/or to performthe localization of the at least one second feature and/or to providethe information about the relative position based on an informationabout an optical property of the at least one optical component. Thisoptical property can be a focal length, e.g. of a single lens of theoptical component, an effective focal length, e.g. of a total opticalsystem of the optical component, a scaling factor, imaging properties,refraction properties, diameters of lenses of the optical component orsurface properties. These properties don't have to be known exactly butcan also be approximated by certain approximations of the opticalproperties or models of the optical component and/or its single opticalelements. For example surface properties of lenses and/or free formlenses and or prisms of the optical component can be described by apolynomial description and/or mesh representation and so on. Therefraction properties of lenses can be described by Snell's law and thatof thin lenses of the optical component can be e.g. approximated by thethin lens approximation. If the optical component comprises more thanone optical element then also different approximations or mathematicaldescriptions for each optical element can be used.

These optical properties of the optical component can be pre-given, e.g.this information is already included in the algorithm used for eyetracking, or this information can be derived by the eye tracking deviceitself, for example, by a calibration procedure. Beneficially, thisinformation about the optical property of the optical component can alsobe used or included in feature search criteria for searching eyefeatures in the image. For example for detecting the pupil in the imagethe eye tracking device searches for circle-like objects or ellipticalobjects having certain dimensions in the image. If now these dimensionsin the image are changed due to the optical component, this can beconsidered when searching for pupil candidates so that the eye trackingdevice searches for circle-like objects or elliptical objects which havedimensions that are rescaled on the basis of the optical property of theat least one optical component. Also forms of objects in the image canchange due to the optical properties of the optical component, so thiscan be considered similarly.

As a result, the at least one first feature and also the localization ofthe at least one second feature can be performed very accurately usinginformation about the optical property of the optical component.Information about the optical property can also be used to provide theinformation about the relative position between the eye and the eyetracking device, especially because this information can be deriveddirectly from the captured images. If the eye tracking device isconfigured to localize eye features in real space using the knowledgeabout the optical property of the optical component, the eye trackingdevice consequently is capable of determining the relative positionbetween the eye and the eye tracking device and moreover also todetermine changes of that relative position. So changes between the headmounted eye tracking device and the user's head do not have any negativeinfluence on the eye tracking performance, in the contrary, by havingmore flexibility in positioning components of the eye tracking deviceand especially performing model based eye tracking the eye trackingquality, e.g. gaze accuracy, can even be enhanced in comparison to knowneye tracking systems.

Furthermore, the head mounted eye tracking device can be configured tomap a position of the at least one second feature in the image and tomodify the mapping in dependency of the information about the relativeposition between the eye and the eye tracking device. Especially, theeye tracking device can map the position of the second feature in theimage to the position of the second feature in real space, e.g.determining the position of the pupil in real space based on theposition of the pupil detected in the image. Also features like the gazedirection and/or point of gaze can be calculated on the basis of theimage and be mapped to the corresponding direction and/or position inreal space. This can be done for example by considering the opticalproperty of the optical component when mapping. On the other hand theposition of the at least one second feature in the image can also bemapped directly to the corresponding value of the first feature. Forexample, the eye tracking device can determine the position of the pupilin the image and map this position to a corresponding gaze direction.For defining this map e.g. a calibration or simulation procedure can beperformed. To achieve that changes in the relative position between theeye tracking device and the eye do not invalidate this map, it is veryadvantageous to modify the mapping in dependency of the informationabout the relative position between the eye and the eye tracking device.This way, the map stays always valid and correctly maps the secondfeature, even if changes in the relative position between the eye andthe eye tracking device take place. These changes can for example bedetected by determining eye corners in the image, additional sensors,like a gyroscope, a proximity sensor, an accelerometer, or an inertialmeasurement unit, or any combination of these sensors and/or by derivingthis information on the basis of the captured images.

The head mounted eye tracking device can also be configured to determinean actual gaze direction from a first gaze direction as the at least onesecond feature of the at least one eye, wherein the information aboutthe optical property of the at least one optical component is used tocorrect the first gaze direction derived from the image to provide theactual gaze direction as the first feature of the eye. Advantageouslythis is a very simple and easy way to determine the gaze direction ofthe user. For example, the eye tracking device is configured todetermine the gaze direction based on the captured images in aconventional way ignoring the influence of the optical component. Thenthis determined first gaze direction can be mapped to the actual gazedirection by a map that considers the optical property of the at leastone optical component. For example this map can be derived by acalibration procedure or be predefined, e.g. as a look-up table. As thismap considers the optical property of the optical component, changes ofthe relative position between the eye tracking device and the eye areconsidered automatically. This also works with other eye featuresdifferent from the gaze direction. So also other features of the eye,like the cornea, pupil position, and so on, can be determined on thebasis of the image and be mapped to the actual features using the mapconsidering the optical property of the optical component.

So it is very advantageous when the head mounted eye tracking device isconfigured to determine the at least one first feature of the at leastone eye and/or to perform the localization of the at least one secondfeature by deriving a property of the at least one second feature of theat least one eye from the image captured by the capturing device whereinthe information about the optical property of the at least one opticalcomponent is used to map the property of the at least one second featurederived from the image to provide the at least one first feature and/orthe localization of the at least one second feature. This property canbe a dimension, position and/or form of the second feature, like acornea diameter or a form of the pupil contour.

Furthermore, the head mounted eye tracking device can be configured toderive the information about the optical property of the at least oneoptical component from calibration results of a calibration procedurefor calibrating the head mounted eye tracking device. If the opticalcomponent comprises a very complex optical system with lenses or prismsand/or free form lenses, that cannot be described in a mathematical wayso easily or the calculation time for calculating the gaze direction onthe basis of complex mathematical formulas describing the opticalcomponent would take too long, it is very beneficial to derive theoptical property from a calibration procedure. This way the opticalproperty of the optical component is implicitly derived from thecalibration procedure and for example on the basis of the calibrationresults a map can be defined which maps the detected or determinedfeatures to the actual ones. Also a model for the optical component canbe used, like a lens model with parameters for a thickness and/or radiusof curvature, wherein these parameters of the model can be fitted duringthe calibration procedure so that the model describes the opticalproperty of the actual optical component approximately or accurately.

Therefore, it is an preferred embodiment of the invention that the headmounted eye tracking device is configured to derive the informationabout the optical property of the at least one optical component from amodel of the at least one optical component, especially wherein themodel of the at least one optical component models the altering of lightpassing through the at least one optical component. As mentioned, thismodel can be used to fit model parameters during a calibrationprocedure. But furthermore, this model can also be pre-given, forexample as an approximation of the optical properties and imagingproperties of the real optical component. If, for example, the opticalcomponent comprises a lens having a thickness much smaller than itsradius of curvature, then the thin-lens approximation might be used tomodel the altering of light passing through the optical component. Alsoother approaches might be used for that model to simplify complex lenssystems and for describing the optical properties of the opticalcomponent.

According to a further embodiment of the invention the head mounted eyetracking device is configured such that, when fixed to the head of auser, a light path from the at least one eye of the user to thecapturing device is altered by the at least one optical component. Soimages of the eye features the capturing device captures are influencedby the optical component. The influence of the optical component on theeye features in the image can be compensated, e.g. by the methodspreviously described. Having the light path from the eye to thecapturing device altered by the at least one optical component or inother words having the optical component in the light path from the eyeto the capturing device has very great advantages with respect to theflexibility of positioning the capturing device. First of all,especially if the head mounted eye tracking device comprises a displaydevice, it is very advantageous to have the optical component as closeas possible to the eye of the user as then the projection of the imagesshown on the displays onto the eyes can be optimized. Also lenses withsmaller diameter can be used for projecting these images to the wholefield of view of the user if these lenses are closer to the eye of theuser. In such a setup it is very hard to position a camera or acapturing device in between the eye and the optical component.Consequently it is very beneficial to be able to position the capturingdevice somewhere behind the optical component and eye tracking can beperformed on the one hand with higher tracking quality and on the otherhand with the possibility of placing the optical component as close aspossible to the eye. Furthermore, by positioning the capturing devicebehind the optical component makes it possible to hide the capturingdevice from the user so that the user is not disturbed by seeing thecapturing device.

Furthermore, the head mounted eye tracking device can comprise at leastone light source for illuminating at least part of the at least one eyewhen the head mounted eye tracking device is fixed to a head of a user.Preferably this light source is configured to cause at least onereflection on the at least one eye. Those reflections, like glints orother Purkinje images, can be used for eye tracking, especially, to moreaccurately determine the positions and orientations of eye features.These one or more light sources can for example emit infrared light,which has the advantage that it cannot be seen by the user so that itdoes not disturb him/her. In this case the capturing device ispreferably configured to be sensitive for at least the infrared spectralrange. The capturing device captures the reflection produced by the atleast one light source and localizes features of the eye using theposition of these reflections with regard to the eye or eye features,e.g. by modeling the cornea as a spherical mirror, which enhances thetracking quality as i.g. explained in the book Remote Eye Tracking byHammond, Chapter 7.

In general, the head mounted eye tracking device can be configured suchthat the light path extends from the at least one light source to the atleast one eye and from the at least one eye to the at least onecapturing device, wherein the at least one optical component ispositioned in that light path, especially so that a light propagatingalong the light path passes through the at least one optical component.So either the capturing device or the at least one light source or evenboth can be placed behind the optical component with regard to a viewingdirection of the user so that the flexibility of optimized positioningof the components of the eye tracking device is enhanced even more.

For the same reason it is preferred that the head mounted eye trackingdevice is configured such that when fixed to the head of a user lightpropagating from the at least one light source to the at least one eyeof the user on the light path is altered in its propagation direction bythe at least one optical component. In this embodiment the light sourceis placed behind the optical component from the perspective of the userso that light emitted by the light source first passes through theoptical component and then hits the eye and produces reflections. Ifalso the capturing device is placed behind the optical component thelight reflected by the eye passes through the optical component againbefore it is captured by the capturing device, so that at least part ofthe light that is captured by the capturing device and constitutes theimage has passed twice through the optical component. Alsoconfigurations are possible in which a capturing device directlycaptures images of the eye without the optical component beingpositioned in between but only the one or more light sources arepositioned behind the optical component so that light emitted by thelight sources passes through the optical component, is reflected by theeye afterwards and then is detected directly by the capturing device. Inthis case only the part of the captured image which relates to thecaused reflections on the eye is influenced by the optical component,wherein other eye features are not influenced. In this case theinformation about the optical property of the optical component is usedfor determining the at least one first feature of the eye only withrespect to the reflections. For example, if point-like and/orcircle-like reflections are produced, the optical component can changethe size and form and position of these reflections, whichadvantageously can be considered when determining the gaze direction orother features. The light sources can also be configured to producestructured reflections or light patterns, e.g. in an annular shape, onthe eye which are then similarly influenced in their size, form orposition by the optical component, which can again considered whendetermining eye features.

According to a further embodiment of the invention the head mounted eyetracking device comprises a planar optical element, preferably a mirror,especially a beam splitting mirror with regard to different spectralranges and/or a dichroic mirror and/or a hot mirror, wherein the headmounted eye tracking device is configured such that, when fixed to thehead of a user, at least part of the light propagating from the at leastone eye of the user to the capturing device is reflected by the planaroptical element, and preferably passes through the at least one opticalcomponent before being reflected by the planar optical element. Thisplanar optical element has the great advantage, that even moreflexibility with regard to the positioning of the capturing device isprovided. The planar optical element is capable of redirecting the lightpath from the eye to the capturing device, so that it is possible toposition the capturing device such, that it cannot be seen by the user.In this context, it is very advantageous to have the planar opticalelement as a dichroic mirror, for example as a hot mirror, which can betransmissive for a predefined spectral range and reflective for secondspectral range, e.g. the infrared spectral range different from thefirst spectral range. This way, when the mirror is transmissive for thevisible wave length range the user can look through the mirror withoutseeing the mirror. At the same time the capturing device can captureinfrared images of the eye that are constituted by light that isreflected by the mirror and which then advantageously can be used fordetermining the at least one first feature of the eye. Alternatively oradditionally, this planar optical element can also be used to redirectlight from the one or more light sources of the eye tracking device forproducing the reflections on the eye of the user. In a similar way alsothese light sources can be hidden from the user and can be optimized intheir positioning.

If the head mounted eye tracking device is configured also as headmounted display the planar optical element can be positioned such andconfigured such that the light originating from the display passesthrough the mirror, through the optical component and then impinges onthe eye so that though the planar optical element is positioned in theview path of the user to the display, he/she can view the image on thedisplay uninterruptedly. Furthermore, the mirror can be positioned onthe optical axis of the optical component and preferably a surfacenormal of the mirror is inclined by an angle, especially 45° degrees,towards the optical axis. This way the optical path is redirected by aright angle by means of the planar optical element.

Furthermore, the head mounted eye tracking device can be configured suchthat, when fixed to the head of a user, light propagating on the lightpath from the at least one light source to the at least one eye of theuser is reflected by the planar optical element, and preferably passesthrough the at least one optical component after being reflected by themirror. So also the light sources can be hidden from the user andnevertheless can be placed in an optimal position for illuminating theeye.

The invention also relates to a method for determining at least onefeature of at least one eye of a user by means of a head mounted eyetracking device with at least one capturing device for capturing lightreflected by the at least one eye and at least one optical component,wherein a propagation direction of light passing through the at leastone optical component is altered. Therein, when the head mounted eyetracking device is fixed to the head of a user, light is captured by theat least one capturing device, wherein the light has passed through theoptical component before and/or after it had been reflected by the atleast one eye of the user and constitutes at least part of an image,wherein the at least one first feature is determined on the basis of theimage and in dependency of an information about a relative positionbetween the at least one eye and the head mounted eye tracking device.

The preferred embodiments and advantages described with regard to thehead mounted eye tracking device according to the inventioncorrespondingly apply to the method according to the invention.Especially, the head mounted eye tracking device according to theinvention can be used for performing the method for determining the atleast one feature with the at least one eye according to the invention.Furthermore, described embodiments of the head mounted eye trackingdevice according to the invention constitute further steps of the methodaccording to the invention.

Even though the invention is described by referring to the at least oneeye of the user, the invention similarly applies to both eyes as well.E.g. the eye tracking device can comprise two capturing units, each forone eye, or a common capturing unit for both eyes, two opticalcomponents, each for one eye, or a common optical component.

Moreover the determination of the at least one first or other featurescan be performed for each eye as described above.

The preferred embodiments refer to the optical component which can befixed in position or dynamic within the head mounted eye trackingdevice. If the optical component comprises several optical elements,each of these optical elements can either be fixed or changeable in it'sposition. The position can be known or unknown according to theexplanations above. Further the at least one optical component or atleast one optical element of the optical component can have a static ordynamic characteristic, e.g. in case of a lens the refractive power canchange over time and space (e.g. in case of a liquid lens), this statecan be known or unknown to the head mounted eye tracking device.

In the following, advantageous embodiments of the present invention aredescribed in more detail with reference to the accompanying drawings.

They show in:

FIG. 1 a schematic illustration of a head mounted eye tracking deviceaccording to a first embodiment of the invention;

FIG. 2 a schematic illustration of a head mounted eye tracking deviceaccording to a second embodiment of the invention;

FIG. 3 a schematic illustration of the principle of ray-tracing used fordescribing the optical properties of the optical component of the headmounted eye tracking device according to an embodiment of the invention;and

FIG. 4 a schematic illustration for reconstructing the eye in a virtualcoordinate system for use in an eye tracking device according to anembodiment of the invention.

FIG. 1 shows a schematic illustration of a head mounted eye trackingdevice 10 a according to a first embodiment of the invention. Generally,the head mounted eye tracking device 10 a comprises a capturing deviceC, which can comprise one or more cameras or sensors for taking picturesof the eye 12 of a user wearing the head mounted eye tracking device 10a. Furthermore, eye tracking device 10 a comprises an optical component14 which can comprise one or more lenses, prisms or other opticalelements. In this example, the optical component 14 comprises a lens E1and optionally further lenses En, which is illustrated by the dashedlens contour in FIG. 1. Also optionally the eye tracking device 10 a cancomprise one or more light sources L, of which two are shown in FIG. 1exemplarily. As further optional components the eye tracking device 10 acan comprise a hot mirror M and a display device 16. Moreover, the eyetracking device 10 a comprises a processing unit 18 for processing thecaptured images and determining at least one feature of the eye 12.

The optical component 14 is placed between the capturing device C andthe user's eye 12 with regard to the optical path from the eye 12 to thecapturing device C so that at least some eye features in the imagecaptured by the capturing device C are altered by the optical component14, e.g. in their form, size and/or position. For example, the lens E1between the capturing device 10 and the eye can magnify the pupil 12 ain the captured image.

The light sources L can produce reflections on the cornea, especially ina structured way, like in an annular form, and/or a point-like wayand/or circle-like way. The light path from the light sources L to theeye 12 is in this setup also altered by the optical component 14.Especially in this configuration the light emitted by the light sourcesL is also reflected by the mirror M, passes through the opticalcomponent 14 and impinges on the eye 12. The images captured by thecapturing device C are processed by the processing unit 18 and featuresof the eye 12 are detected.

The hot mirror M facilitates more flexibility with regard to thecapturing device C and the light sources L and still makes a centralview of the capturing device C onto to the eye possible. This isillustrated by the virtual camera Cv. The view of the capturing device Ccorresponds to the view of a camera at the position of the virtualcamera Cv without the mirror M.

In other embodiments of the invention, for example for a different useof the head mounted eye tracking device 10 a, e.g. for medical ordiagnosis purpose, the capturing device C could also be placed in theposition of the shown virtual camera Cv and the mirror M and the displaydevice 16 can be omitted. The capturing device C can also comprise morethan one camera or sensors in different places. The capturing unit Ccould also be placed to have direct view onto the eye 12 without havingthe optical component 14 in between and only the light sources L areplaced such that the light path from the light sources L to the eye 12passes through the optical component 14. On the other hand, also thelight sources L could be placed such, that they illuminate the lightdirectly without having the optical component 14 in between and thecapturing device C is positioned as shown.

The light sources L and/or the capturing device C can even be placedbetween elements of the optical component 14. So there are manypossibilities for optimal positioning of the components of the eyetracking device 10 a by which the optical properties and the eyetracking quality can be optimized.

FIG. 2 shows a schematic illustration of a head mounted eye trackingdevice 10 b according to another embodiment of the invention. In thisembodiment an optical component 14 comprises a free-form lens E andlight captured by the capturing device C propagates from the eye 12through the free-form lens E, is reflected by the hot mirror M and thencaptured by the capturing device C. In general any kind of opticalwaveguide can be used additionally to or instead of this free-form lensE. During the light is propagating through the optical component 14 itis several times internally reflected by the surface of the free-formlens E, for which purpose the lens E optionally can comprise areflective coating on parts of its surface. Furthermore, here again theeye tracking device 10 b can comprise a display unit 16, wherein lightfrom the display unit 16 passes through the hot mirror M, through thefree-form lens E and finally impinges on the eye 12. Here also the eyetracking device 10 b can optionally comprise one or more light sources,which are not shown in this case. These optional light sources can bepositioned so that they illuminate the eye 12 directly and/or throughthe free-form lens E. Using a free-form lens E has the advantage thatthe eye tracking device 10 b can be built even more compact and at thesame time components of the eye tracking device like the capturingdevice C can be hidden from the user.

In these situations when the capturing device C captures light that haspassed through the optical component 14 to constitute at least part ofthe image of the eye 12, different from state of the art eye trackingtechniques, the processing unit 18 now has to deal with the fact thatthe observed eye 12, or the observed glints, in the image is not adirect projection of the real eye 12 onto the sensors of the capturingdevice C, but maybe altered by the optical component 14. In order tocompensate for this, several different techniques can be applied, whichare explained in more detail in the following with regard to a setup, inwhich the capturing device C is placed behind the optical component 14but which applies for light sources L being placed behind the opticalcomponent 14 as well.

First of all, the optical properties of the optical component 14 can betaken into account implicitly by performing a calibration procedure andbased on this calibration a map can be defined which maps the positionsof certain eye features in the captured images, like the position of thepupil, to the corresponding real eye features, like the position of thepupil 12 a in real space, or to other eye features like the gazedirection. The general problem with this approach is that once the usermoves his/her head relative to the eye tracking device 10 a, 10 b, forexample if the head mounted eye tracking device 10 a, 10 b slightlyslips, then eye tracking would not work accurately anymore. Theinvention advantageously solves this problem by taking into account atleast one information about relative position between the eye trackingdevice 10 a, 10 b and the eye 12 when determining features of the eye12. This can be done for example by detecting eye corners 12 b or otherfeatures that do not move with regard to the eye tracking device 10 b ifthe user changes his/her gaze direction except the eye tracking device10 a, 10 b changes its position with respect to the user's head. Thismovement would lead to images in which the captured eye corners 12 balso comprise a different position from that of images taken before. Forexample the position shift of the eye corners 12 b can be determined inthe images and used for shifting back the detected position of the pupil12 a in the image.

Also other methods like raytracing, reconstructing the eyes in virtualcoordinate systems, undistorting the camera image, using a virtualcamera and/or reconstructing the gaze on a virtual stimulus plane can beused and are explained in the following.

Most of these methods use a model of the optical component 14, but thisis not a necessity. There are several models for the optical component14 or parts thereof, like models for a lens or lens systems or otheroptical elements as part of the optical component 14, and this inventiondoes not rely on a specific one. Any model which describes therefracting properties of the optical component 14 or its elements orapproximates them can be used. For example, the optical component 14 canbe modeled as a set or a combination of single elements, wherein eachelement can be described by a paraxial lens model using the paraxialapproximation, especially for thin lenses, a spherical lens model, athick lens model and/or a free-form model. A free-form model comprises aclosed surface, which can be represented in terms of elementary shapeslike ellipsoid, plane, box, paraboloid, and so on, which are combinedusing Constructive Solid Geometry laws, like intersection, subtraction,union, clipping of multiple shapes and so on, as well as in terms ofthrough tessellation, linearization, approximation with a mesh oftriangles or polygons or second order surfaces. But it may also bepossible to describe the whole optical system of the optical component14 or a subset of the optical component 14 with a simplified model or,if this is not available, to rely on pre-computed or otherwise existingrepresentations of the light path through the optical component 14, likea raytracing for one or more specific camera poses with respect to theoptical component 14.

FIG. 3 shows a schematic illustration of the principle of raytracing foruse in a head mounted eye tracking device 10 a, 10 b according to anembodiment of the invention, especially for taking into account theoptical properties of the optical component 14. An idea of this methodis to trace the rays back from the capturing unit C, e.g. a camera, orof a light source, which is denoted with a reference sign B in FIG. 3,into the direction Dir1 of the detected features, that are representedby the observed point P1 in FIG. 3, until they hit the optical component14. Then, the ray at the optical component 14 is refracted and oneobtains a new outgoing ray, especially a refracted ray, which can bedescribed by a point P2 on that ray and its new direction Dir2. How todo the refraction depends on the model of the optical component 14, e.g.on a lens model. The raytracing can also be beneficial in the otherdirection when one would be interested in the refracted image of a knownpoint on e.g. the camera sensor. For example for thin lenses of theoptical component 14 the thin lens approximation can be used formodeling the light refraction. Furthermore, for the capturing unit C apinhole camera model can be used, meaning the Capturing unit comprisescamera with a camera sensor and an aperture that is assumed to have aninfinitesimal opening, so that for each point and/or pixel of the camerasensor one light direction can be assigned, from which light has passedthrough the aperture and had impinged on that point and/pixel. In anembodiment the thin-lens approximation is used for the optical component14 to do a bidirectional correction of rays in order to correct lightrays derived from e.g. the pinhole camera model as well as to project 3Dpoints to the camera sensor in a way as it would be perceived by thecamera when observing that point through the at least one opticalcomponent 14.

In some embodiments, the position and orientation in space of thecapturing device C, e.g. the camera, of each lens element E1, En, E ofthe optical component 14, of each light source L and eventually of otheroptical elements such as mirrors M are known. In other embodiments theposition of some or all of such elements might not be known and in thatcase a calibration procedure can be performed in order to find theunknown values of said parameters which minimize the overall calibrationerror. Hereby, the calibration is not necessarily explicit, i.e. notnecessarily performed by building a model of the components, but canalso be implicit by using a global mapping, like a homography or evenbeing determined and/or derived and/or decomposed from a usercalibration.

The coordinates of points and/or directions in the real word (meaningmetric points) are determined with the help of known or calibratedand/or mapped coordinates and properties of the optical setup by meansof a applying a series of raytracing transformations. Thesetransformations can be performed by applying the law of reflection atreflective surfaces, that is the angle of incidence with respect to thesurface normal and the angle of reflection are the same, and/or byapplying Snell's law at refractive surfaces. That refracted ray can beused in an algorithm which reconstructs the eye 12 or features of theeye 12 instead of using the original ray coming from the capturingdevice C, e.g. a camera or image sensor.

In the case that the eye tracking device 10 a, 10 b comprises one ormultiple light sources L, the assumed direct light path causing forexample a reflection on the cornea is corrected by the describedraytracing. This covers all possible setups of cameras, light sources L,mirrors and the eye 12 with respect to the optical component 14, e.g. alens system. For example, capturing device C, e.g. the camera, can be onone side, the eye 12 and the light sources L on the other side. But thecamera and the light sources L could also be on the same side or evensomewhere in the middle of the lens system of the optical component 14.

In general three types of raytracing can be performed. The first isbackward raytracing, which is done by originating from the coordinatesof the points, for example, a glint or cornea reflection, on a cameraimage plane, e.g. using the pinhole camera model. Such a ray is thencast until it hits the surface of components of the optical component 14and goes through a chain of refractions and reflections. Second isforward raytracing for the rays originating directly from the lightsources L or features of the eye 12, and after a chain of refractionsand reflections hitting the camera's image sensor. A third is mixedforward and backward raytracing, wherein rays coming from the lightsources L and rays which correspond to seen features in the image areconsidered and used to find the parameters of the eye model.

In theory, this approach can be used for almost all eye trackingalgorithms which so far do not consider having optical components 14 inthe path to the user by replacing rays coming from the camera by rayswhich are altered by the optical component 14. As an example, we show anapproach of using raytracing to reconstruct the real eye position andgaze direction. The idea of doing backwards raytracing is more generaland not only limited to this algorithm.

As an example, it is possible to consider a set of possible eyestates/hypotheses (e.g. particle filters) where each state hypothesiscontains the full set of coordinates of the eye components which areincluded in the model, for example eye position and orientation,including visual axis to optical axis shift and so on. It can be thenfor each said hypothesis been performed a raytracing for each featurewhich is expected to be seen in the image. Such feature can be the pupilcenter and/or contour, iris center and/or contour, eye corners, eyelids, cornea reflections etc. So the expected location of said featuresin the (virtual) image can be found. The likelihood of said hypothesiscan then be computed as a function of the distance of each of itsfeatures (in 2D) from the detected features in the real image, applyinga statistical measurement error model.

FIG. 4 shows a schematic illustration of reconstructing the eye invirtual coordinate system for use in a head mounted eye tracking device10 a, 10 b according to an embodiment of the invention, especially fortaking into account the optical properties of the optical component 14.In this example the optical component 14 is exemplarily configured as athin lens E2. When looking with the camera through this lens E2 onto auser's eye, the camera will see a virtual image of such when positionedwithin the focal length of the lens E2 of the head mounted eye trackingdevice 10 a, 10 b or head mounted display. This is illustrated in FIG.4. Here, 12 c denotes the limbus of the real eye 12 and E2 the thin lensin thin lens approximation. Furthermore, f denotes the focal length ofthe lens E2, wherein the focal length f is larger than the objectdistance g so that an image of the real limbus 12 c is produced as avirtual image 12 c′ at the image distance b.

Any eye tracking method which is based on capturing images of the eyecan be applied to that image ignoring that the observed eye is distortedby the lens, or in general an optical system, of the optical component14. The output of such algorithm will be eye parameters, like positionand/or orientation of the eye, gaze and so on, which are not reflectingthe true state of the eye, but describe a virtual eye. The goal of thismethod is to calculate corresponding real eye parameters from observedvirtual parameters. A principle is to take any point of thereconstructed eye, for example the eye center, cornea position, pupiland/or limbus center, contour point of the limbus, and so on, in thevirtual coordinates and transform them to real coordinates. Thetransformation is fully defined by the optical component 14, e.g. by thelens or lens system thereof, and can be for example implemented as alook-up table, (non-) linear mapping, or interpolations. Either thetransformation can be explicitly computed like for a simple thin lensmodel, or it can be obtained numerically, for example, by performing anoff-line raytracing simulation of the whole optical system of theoptical component 14.

One possibility is to define a range of possible eyeball locations andeye orientations. This set can be sampled by a grid of finite locationsand/or orientations, with arbitrary spacing and distribution. Raytracingis then performed for each sample and the coordinates of each relevanteye feature are stored in a look-up table against the ground truthcoordinates of the simulation. For example, when having this look-uptable one could look up 2D cornea reflections determined on the imagesensor, which sees a virtual image of the eye, and get the corresponding3D coordinates of the cornea center, in real metric coordinates.

Another method is to undistort the camera image. The virtual image of auser's eye which is seen through the optical component 14, e.g. througha lens or lens system, is mathematically related to the real image onewould see if there would be no optical component 14, e.g. lens or lenssystem, in between, at least for a known or assumed distance of the eyeform the optical component 14 which can be determined using othersensors like proximity sensors. The goal is to undistort this image,that is to calculated corresponding real 2D-points from observed virtual2D-points. The principle is to determine a directly observable eyefeature, like the limbus, or part of a directly visible eye feature,like the contour point of the limbus, in the virtual eye image and tocorrect the point (or direction) afterwards using the properties of theone or more lenses of the optical component 14 or other optical elementsof the optical component 14.

Another method would be to use a virtual camera. Instead of correctingthe observed virtual image by the real, for example pinhole, camera, onecould construct a virtual camera which models the optical component 14,e.g. lenses, of the head mounted eye tracking device 10 a, 10 b as partof the camera.

Another possibility is to use such simulation to create an approximatevirtual camera representing the complex optical setup, described by apredefined set of parameters, which may include virtual 6D coordinatesof the virtual camera, field of view horizontal and vertical, principalaxis shift, tilt of the image plane with respect to the optical axis ofthe virtual camera. In addition, it can be computed a 2D distortionmodel of the virtual camera, which can have polar components,polynomial, or other non-linear model. Alternatively the raytracing canbe done just to pre-compute a table or to learn the parameters of amodel or function, which maps the relative 2D coordinates between theeye feature (for example 2D pupil center to 2D cornea reflectionscenter) to 3D eye position and/or orientation of the eye.

A further method is to reconstruct the gaze on a virtual stimulus plane.Instead of correcting for the optical component 14, e.g. the lens, oneignores the optical component 14 and eye tracking is done viatraditional algorithm (not including a lens). This means the gaze iscalculated based on the altered image and the final output of the eyefeatures (for example the gaze) is corrected afterwards.

This method can be an simple work-around to achieve valid gaze datawithout dealing too much with the lens. Furthermore, the approach is notlimited in the number of optical elements of the optical component 14.First, eye tracking with the optical component 14 in front of the camerais performed and the eye position and gaze orientation, as well as allother relevant features are reconstructed ignoring the existence of theoptical component 14. If necessary for later steps the calculated‘virtual’ parameters can be mapped, using a possibly predeterminedrelation, into ‘real’ coordinates.

The transformation can be done either directly with vector algebrataking the properties of the lens into account. Another way can be tocompute the mapping of real to virtual coordinates, to performing atessellation of the virtual stimulus plane by direct raytracing and thento apply an interpolation. This method can be applied also when theoptical component 14 comprises more than one lens or even a complexoptical system.

All in all the invention makes it possible to gain flexibility inpositioning the capturing device and illumination sources within thehead mounted eye tracking device and to hide the components from theuser. Furthermore a capturing device of the eye tracking device can bepositioned so that feature visibility of eye features is optimized overdifferent eye positions and motions.

1-15. (canceled)
 16. A method comprising: capturing a first image of aneye of a user at a first time; determining a first gaze direction of theeye based on the first image; capturing a second image of the eye of theuser at a second time; determining a change in position of agaze-invariant feature of the user from the first time to the secondtime; and determining a second gaze direction of the eye based on thesecond image and the change in position of the gaze-invariant feature.17. The method of claim 16, wherein determining the change in positionof the gaze-invariant feature includes determining a change in positionof the gaze-invariant feature in the second image as compared to thefirst image.
 18. The method of claim 16, wherein determining the changein position of the gaze-invariant feature includes receiving data from amotion sensor.
 19. The method of claim 16, further comprisingdetermining a first position of a gaze-variant feature of the user basedon the first image, wherein determining the first gaze direction isbased on the first position of the gaze-variant feature.
 20. The methodof claim 19, further comprising: determining a second position of thegaze-variant feature of the user based on the second image; anddetermining a corrected position of the gaze-variant feature of the userbased on the second position of the gaze-variant feature and the changein position of the gaze-invariant feature, wherein determining thesecond gaze direction is based on the corrected position of thegaze-variant feature.
 21. The method of claim 20, wherein determiningthe second gaze direction based on the corrected position includesmodeling the refractive properties of at least one refractive opticalelement.
 22. The method of claim 20, wherein the gaze-variant featureincludes at least one of a pupil center or an iris center.
 23. Themethod of claim 16, wherein the gaze-invariant feature includes at leastone of an eye corner, and eyebrow, or a nose.
 24. The method of claim16, further comprising: capturing a third image of a second eye of theuser at the first time; determining a third gaze direction of the secondeye based on the third image; capturing a fourth image of a second eyeof the user at the second time; and determining a fourth gaze directionof the second eye based on the fourth image and the change in positionof the gaze-invariant feature.
 25. The method of claim 24, furthercomprising determining a point of gaze based on the second gazedirection of the eye and the fourth gaze direction of the second eye.26. An apparatus comprising: a camera to capturing a first image of aneye of a user at a first time and capture a second image of the eye ofthe user at a second time; and a processor to: determine a first gazedirection of the eye based on the first image; determine a change inposition of a gaze-invariant feature of the user; and determine a secondgaze direction of the eye based on the second image and the change inposition of the gaze-invariant feature.
 27. The apparatus of claim 26,wherein the processor is to determine a first position of a gaze-variantfeature of the user based on the first image and determine the firstgaze direction based on the first position of the gaze-variant feature;wherein the processor is to determine a second position of thegaze-variant feature of the user based on the second image; wherein theprocessor is to determine a corrected position of the gaze-variantfeature of the user based on the second position of the gaze-variantfeature and the change in position of the gaze-invariant feature; andwherein the processor is to determine the second gaze direction based onthe corrected position of the gaze-variant feature.
 28. The apparatus ofclaim 27, wherein the processor is to determine the second gazedirection based on the corrected position by modeling the refractiveproperties of at least one refractive optical element.
 29. The apparatusof claim 27, wherein the gaze-variant feature includes at least one of apupil center or an iris center.
 30. The apparatus of claim 26, whereinthe gaze-invariant feature includes at least one of an eye corner, andeyebrow, or a nose.
 31. The apparatus of claim 26, wherein the camera isfurther to capture a third image of a second eye of the user at thefirst time and capture a fourth image of a second eye of the user at thesecond time, wherein the processor is to determine a third gazedirection of the second eye based on the third image and determine afourth gaze direction of the second eye based on the fourth image andthe change in position of the gaze-invariant feature.
 32. Anon-transitory computer-readable medium encoding instructions which,when executed by a processor, cause a processor to perform operationscomprising: determining a first gaze direction of the eye based on afirst image of an eye of a user taken at a first time; determining achange in position of a gaze-invariant feature of the user; anddetermining a second gaze direction of the eye based on the change inposition of the gaze-invariant feature of the user and a second image ofthe eye of the user taken at a second time.
 33. The non-transitorycomputer-readable medium of claim 32, wherein determining the change inposition of the gaze-invariant feature of the user includes determininga change in position of the gaze-invariant feature in the second imageas compared to the first image.
 34. The non-transitory computer-readablemedium of claim 32, wherein the operations further comprise: determininga first position of a gaze-variant feature of the user based on thefirst image, wherein determining the first gaze direction is based onthe first position of the gaze-variant feature; determining a secondposition of the gaze-variant feature of the user based on the secondimage; and determining a corrected position of the gaze-variant featureof the user based on the second position of the gaze-variant feature andthe change in position of the gaze-invariant feature, whereindetermining the second gaze direction is based on the corrected positionof the gaze-variant feature.
 35. The non-transitory computer-readablemedium of claim 32, wherein determining the second gaze direction basedon the corrected position includes modeling the refractive properties ofat least one refractive optical element.